48 research outputs found
Deep learning systems as complex networks
Thanks to the availability of large scale digital datasets and massive
amounts of computational power, deep learning algorithms can learn
representations of data by exploiting multiple levels of abstraction. These
machine learning methods have greatly improved the state-of-the-art in many
challenging cognitive tasks, such as visual object recognition, speech
processing, natural language understanding and automatic translation. In
particular, one class of deep learning models, known as deep belief networks,
can discover intricate statistical structure in large data sets in a completely
unsupervised fashion, by learning a generative model of the data using
Hebbian-like learning mechanisms. Although these self-organizing systems can be
conveniently formalized within the framework of statistical mechanics, their
internal functioning remains opaque, because their emergent dynamics cannot be
solved analytically. In this article we propose to study deep belief networks
using techniques commonly employed in the study of complex networks, in order
to gain some insights into the structural and functional properties of the
computational graph resulting from the learning process.Comment: 20 pages, 9 figure
Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning models
Creating learning models that can exhibit sophisticated reasoning skills is
one of the greatest challenges in deep learning research, and mathematics is
rapidly becoming one of the target domains for assessing scientific progress in
this direction. In the past few years there has been an explosion of neural
network architectures, data sets, and benchmarks specifically designed to
tackle mathematical problems, reporting notable success in disparate fields
such as automated theorem proving, numerical integration, and discovery of new
conjectures or matrix multiplication algorithms. However, despite these
impressive achievements it is still unclear whether deep learning models
possess an elementary understanding of quantities and symbolic numbers. In this
survey we critically examine the recent literature, concluding that even
state-of-the-art architectures often fall short when probed with relatively
simple tasks designed to test basic numerical and arithmetic knowledge
Modeling cognition with generative neural networks: The case of orthographic processing
This thesis investigates the potential of generative neural networks to model cognitive processes. In contrast to many popular connectionist models, the computational framework adopted in this research work emphasizes the generative nature of cognition, suggesting that one of the primary goals of cognitive systems is to learn an internal model of the surrounding environment that can be used to infer causes and make predictions about the upcoming sensory information. In particular, we consider a powerful class of recurrent neural networks that learn probabilistic generative models from experience in a completely unsupervised way, by extracting high-order statistical structure from a set of observed variables. Notably, this type of networks can be conveniently formalized within the more general framework of probabilistic graphical models, which provides a unified language to describe both neural networks and structured Bayesian models. Moreover, recent advances allow to extend basic network architectures to build more powerful systems, which exploit multiple processing stages to perform learning and inference over hierarchical models, or which exploit delayed recurrent connections to process sequential information. We argue that these advanced network architectures constitute a promising alternative to the more traditional, feed-forward, supervised neural networks, because they more neatly capture the functional and structural organization of cortical circuits, providing a principled way to combine top-down, high-level contextual information with bottom-up, sensory evidence. We provide empirical support justifying the use of these models by studying how efficient implementations of hierarchical and temporal generative networks can extract information from large datasets containing thousands of patterns. In particular, we perform computational simulations of recognition of handwritten and printed characters belonging to different writing scripts, which are successively combined spatially or temporally in order to build more complex orthographic units such as those constituting English words
Emergence of Network Motifs in Deep Neural Networks
Network science can offer fundamental insights into the structural and
functional properties of complex systems. For example, it is widely known that
neuronal circuits tend to organize into basic functional topological modules,
called "network motifs". In this article we show that network science tools can
be successfully applied also to the study of artificial neural networks
operating according to self-organizing (learning) principles. In particular, we
study the emergence of network motifs in multi-layer perceptrons, whose initial
connectivity is defined as a stack of fully-connected, bipartite graphs. Our
simulations show that the final network topology is primarily shaped by
learning dynamics, but can be strongly biased by choosing appropriate weight
initialization schemes. Overall, our results suggest that non-trivial
initialization strategies can make learning more effective by promoting the
development of useful network motifs, which are often surprisingly consistent
with those observed in general transduction networks
Cognition-Based Networks: A New Perspective on Network Optimization Using Learning and Distributed Intelligence
IEEE Access
Volume 3, 2015, Article number 7217798, Pages 1512-1530
Open Access
Cognition-based networks: A new perspective on network optimization using learning and distributed intelligence (Article)
Zorzi, M.a , Zanella, A.a, Testolin, A.b, De Filippo De Grazia, M.b, Zorzi, M.bc
a Department of Information Engineering, University of Padua, Padua, Italy
b Department of General Psychology, University of Padua, Padua, Italy
c IRCCS San Camillo Foundation, Venice-Lido, Italy
View additional affiliations
View references (107)
Abstract
In response to the new challenges in the design and operation of communication networks, and taking inspiration from how living beings deal with complexity and scalability, in this paper we introduce an innovative system concept called COgnition-BAsed NETworkS (COBANETS). The proposed approach develops around the systematic application of advanced machine learning techniques and, in particular, unsupervised deep learning and probabilistic generative models for system-wide learning, modeling, optimization, and data representation. Moreover, in COBANETS, we propose to combine this learning architecture with the emerging network virtualization paradigms, which make it possible to actuate automatic optimization and reconfiguration strategies at the system level, thus fully unleashing the potential of the learning approach. Compared with the past and current research efforts in this area, the technical approach outlined in this paper is deeply interdisciplinary and more comprehensive, calling for the synergic combination of expertise of computer scientists, communications and networking engineers, and cognitive scientists, with the ultimate aim of breaking new ground through a profound rethinking of how the modern understanding of cognition can be used in the management and optimization of telecommunication network
Automated Detection of Dolphin Whistles with Convolutional Networks and Transfer Learning
Effective conservation of maritime environments and wildlife management of
endangered species require the implementation of efficient, accurate and
scalable solutions for environmental monitoring. Ecoacoustics offers the
advantages of non-invasive, long-duration sampling of environmental sounds and
has the potential to become the reference tool for biodiversity surveying.
However, the analysis and interpretation of acoustic data is a time-consuming
process that often requires a great amount of human supervision. This issue
might be tackled by exploiting modern techniques for automatic audio signal
analysis, which have recently achieved impressive performance thanks to the
advances in deep learning research. In this paper we show that convolutional
neural networks can indeed significantly outperform traditional automatic
methods in a challenging detection task: identification of dolphin whistles
from underwater audio recordings. The proposed system can detect signals even
in the presence of ambient noise, at the same time consistently reducing the
likelihood of producing false positives and false negatives. Our results
further support the adoption of artificial intelligence technology to improve
the automatic monitoring of marine ecosystems
The role of architectural and learning constraints in neural network models: A case study on visual space coding
The recent “deep learning revolution” in artificial neural networks had strong impact and widespread deployment for engineering applications, but the use of deep learning for neurocomputational modeling has been so far limited. In this article we argue that unsupervised deep learning represents an important step forward for improving neurocomputational models of perception and cognition, because it emphasizes the role of generative learning as opposed to discriminative (supervised) learning. As a case study, we present a series of simulations investigating the emergence of neural coding of visual space for sensorimotor transformations. We compare different network architectures commonly used as building blocks for unsupervised deep learning by systematically testing the type of receptive fields and gain modulation developed by the hidden neurons. In particular, we compare Restricted Boltzmann Machines (RBMs), which are stochastic, generative networks with bidirectional connections trained using contrastive divergence, with autoencoders, which are deterministic networks trained using error backpropagation. For both learning architectures we also explore the role of sparse coding, which has been identified as a fundamental principle of neural computation. The unsupervised models are then compared with supervised, feed-forward networks that learn an explicit mapping between different spatial reference frames. Our simulations show that both architectural and learning constraints strongly influenced the emergent coding of visual space in terms of distribution of tuning functions at the level of single neurons. Unsupervised models, and particularly RBMs, were found to more closely adhere to neurophysiological data from single-cell recordings in the primate parietal cortex. These results provide new insights into how basic properties of artificial neural networks might be relevant for modeling neural information processing in biological systems
Machine Learning-aided Design of Thinned Antenna Arrays for Optimized Network Level Performance
With the advent of millimeter wave (mmWave) communications, the combination
of a detailed 5G network simulator with an accurate antenna radiation model is
required to analyze the realistic performance of complex cellular scenarios.
However, due to the complexity of both electromagnetic and network models, the
design and optimization of antenna arrays is generally infeasible due to the
required computational resources and simulation time. In this paper, we propose
a Machine Learning framework that enables a simulation-based optimization of
the antenna design. We show how learning methods are able to emulate a complex
simulator with a modest dataset obtained from it, enabling a global numerical
optimization over a vast multi-dimensional parameter space in a reasonable
amount of time. Overall, our results show that the proposed methodology can be
successfully applied to the optimization of thinned antenna arrays.Comment: 5 pages, 7 figures. This paper has been presented at EuCAP 2020.
Copyright IEEE 2020. Please cite it as: M. Lecci, P. Testolina, M. Rebato, A.
Testolin, and M. Zorzi, "Machine Learning-aided Design of Thinned Antenna
Arrays for Optimized Network Level Performance," 14th European Conference on
Antennas and Propagation (EuCAP 2020), Copenhagen, Mar. 202